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THE  RELATIVE  EFFECTIVENESS  OF  DIFFERENT  METHODS  OF 
STATING  IMAGE  INTERPRETATION  REQUIREMENTS 


The  Army  Image  Interpreter  working  in  a  tactical  environment  must 
be  responsive  to  a  number  of  interpretation  requirements.  Foremost 
among  these  requirements  are  those  for  accuracy  and  completeness  of 
information.  Problems  arise  when  these  requirements  are  imposed 
simultaneously.  For  example,  if  the  requirement  is  for  completeness 
of  target  identification  (that  is,  all  targets  must  be  identified), 
interpreters  may  respond  to  doubtful  cues  and  signatures,  and  achieve 
completeness  at  the  sacrifice  of  accuracy.  On  the  other  hand,  a 
requirement  for  high  accuracy  may  lead  interpreters  to  identify  only 
the  more  discriminable  targets,  and  to  ignore  doubtful  ones.  Simply 
instructing  interpreters  to  be  accurate  and  complete  in  all  situations 
may  not  lead  to  the  optimal  information  output  for  any  particular 
situation.  / 

V\ 

Edwards--  lias  proposed  a  solution  to  problems  where  the  instructions 
may  be  ambiguous  or  may  require  working  in  accordance  with  conflicting 
purposes.  His  solution  proposes  setting  up  a  payoff  matrix  and  assigning 
values  to  each  response  as  it  relates  to  the  true  state.  For  example, 
identifying  a  tank  as  a  truck  vs.  identifying  a  jeep  as  a  tank.  From 
the  assigned  costs  and  values,  the  interpreter  can  determine  the  approach 
which  will  maximize  his  performance  in  terms  of  his  specific  objective. 

More  recently  signal  detection  theorists  have  utilized  the  payoff 
matrix  approach  to  obtain  differential  performance  in  the  detection  of 
signals.  Thomas  and  Sadacca,  ^ using  a  modified  payoff  approach,  found 
that  interpreters  can  vary  their  performance  in  accordance  with  payoff 
instructions.  In  that  study,  however,  the  instructions  to  interpreters 
were  a  combination  of  the  payoff,  certitude,  and  verbal  methods  described 
below.  The  present  study  is  concerned  with  the  relative  effectiveness 
of  the  three  methods  of  stating  instructions.  The  specific  objective 
was  to  determine  how  the  three  methods  of  stating  information  require¬ 
ments  affect  the  accuracy  and  completeness  of  interpretation  under  each 
of  three  types  of  requirement --for  accuracy,  for  completeness,  and  for 
accuracy  a.id  completeness  combined. 


Edwards ,  Ward .  Payoffs  are  Instructions.  Psychological  Review.  68, 
1961.  p  275-284. 

^ Thomas ,  J.  A.  and  Sadacca,  R.  Ability  of  image  interpreters  to  adapt 
output  to  varying  requirements  for  completeness  and  accuracy.  Techni¬ 
cal  Research  Note  165.  U.  S.  Army  Personnel  Research  Office.  Decem¬ 
ber  1965* 


GENERAL  APPROACH 


INTERPRETATION  REQUIREMENTS 

The  information  requirements  differed  in  the  emphasis  placed  on 
three  attributes: 

1.  Accuracy  of  target  identification. 

2.  Completeness  of  target  identification. 

3.  Accuracy  and  completeness  of  target  identification  combined. 


METHODS  OF  STATING  REQUIREMENTS 

Payoff  method .  This  method  is  a  modification  of  the  payoff  matrix 
approach  suggested  by  Edwards  and  others.  Under  this  method,  the  inter¬ 
preter  is  advised  of  the  number  of  points  which  will  be  awarded  for  each 
correct  identification  or  deducted  as  a  penalty  for  each  incorrect 
identification.  In  the  present  study,  the  interpreter  was  not  told  to 
be  complete  in  his  interpretation,  or  to  be  accurate,  or  to  be  complete 
and  accurate.  Instead,  for  each  requirement,  he  was  given  a  scoring 
formula--the  payoff — for  correct  and  incorrect  responses  to  serve  as 
the  rule  under  which  he  was  to  operate. 

For  the  accuracy  requirement,  the  instructions  stated  that  one 
point  would  be  awarded  for  each  correct  identification  and  nine  points 
would  be  deducted  for  each  incorrect  identification. 

For  the  completeness  requirement,  the  interpreter  was  informed 
that  he  would  be  given  one  point  for  each  correct  identification,  and 
that  no  points  would  be  subtracted  for  each  incorrect  identification. 

For  the  accuracy  and  completeness  requirement,  the  instructions 
stated  that  one  point  would  be  awarded  for  each  correct  identification, 
and  one  point  subtracted  for  each  incorrect  identification. 

Certitude  Method .  The  interpreter  was  instructed  to  identify  only 
those  targets  for  which  his  level  of  confidence  was  above  a  stipulated 
figure.  Level  of  confidence  is  an  expression  of  the  interpreter's 
feeling  of  certitude  in  his  identification.  For  example,  a  confidence 
level  of  90#  indicates  that  the  interpreter  feels  that  he  has  a  90  per¬ 
cent  or  a  9  bo  1  chance  of  being  correct.  Interpreters  had  been  previously 
instructed  in  the  use  of  levels  of  confidence. 

Under  the  accuracy  requirement,  the  interpreter  was  instructed  to 
"only  identify  those  targets  for  which  your  level  of  confidence  is  90$ 
or  better". 

Under  the  completeness  requirement,  the  interpreter  was  instructed 
to  "identify  all  the  targets  you  possibly  can,  even  if  your  level  of 
confidence  may  be  near  zero" . 


-2- 


Under  the  accuracy  and  completeness  requirement ,  the  interpreter 
was  instructed  to  "only  identify  those  targets  for  which  your  level  of 
confidence  is  50$  or  better". 

Verbal  Method.  Under  this  method,  the  requirements  were  presented 
to  the  interpreter  us  verbal  summary  statements  of  the  particular  require¬ 
ment. 


Under  the  accuracy  requirement,  the  interpreter  was  instructed  to 
"be  as  highly  accurate  as  possible  in  identifying  the  tactical  targets". 

For  the  completeness  requirement,  the  interpreter  was  instructed 
to  "be  as  complete  as  you  can  in  identifying  tactical  targets". 

Under  the  accuracy  and  completeness  requirement,  the  interpreter 
was  instructed  to  "be  as  complete  and  as  accurate  as  possible  in 
identifying  tactical  targets  . . " 

A  complete  set  of  instructions  used  under  the  three  methods  is 
included  as  Appendix  A.  The  methods  and  the  instructions  for  each 
method  used  for  each  requirement  is  summarized  in  Table  1. 

Table  1 

SUMMARY  OF  PROCEDURES  EMPLOYED  IN  EXPERIMENT 


Method  of 
Stating 
Requirements 


Value 

Statement  for 
Response 
Requirements 


Image  Interpretation  Requirements 


Completeness 

(T-17) 


Completeness - 
Accuracy  (T-10) 


Accuracy 

(T-23) 


Payoff 


Score  for 
rights 


rights  =  +1 


rights  =  +1 


rights  =  +1 


Score  for 
wrongs 


wrongs  =  0 


wrongs  =«  -1 


wrongs  =  -9 


Certitude 


Minimum  accept¬ 
able  level  of 
certitude 


0$ 


50$ 


90$ 


Verbal 


Description 

(Verbal) 


Be  very 
coiiqilcte 


Be  accurate 
and  complete 


Be  very 
accurate 


EXPERIMENTAL  METHOD 


SUBJECT 

Thirty  image  interpreter  trainees  in  the  last  week  of  training  at 
the  U.  S.  Army  Intelligence  School,  Ft.  Ilolabird,  Maryland,  were  used 
in  this  experiment.  The  sample  was  divided  into  three  matched  groups 
of  ten  each.  The  matching  procedure  vns  accomplished  on  the  basis  of 
Rights  and  Wrongs  scores  on  three  standard  performance  measures. 


PERFORMANCE  MEASURES 

A  scries  of  3  tactical  performance  measures,  representing  typical 
real-life  work  situations  which  confront  the  image  interpreter,  was 
selected  for  this  experiment,  T-10,  T-17,  and  T-23.  Imagery  for  the 
performance  measures  was  selected  from  actual  tactical  and  strategic 
air  reconnaissance  missions.  Some  photographs,  taken  during  wartime, 
showed  enemy-held  positions. 

As  in  the  operational  situation,  the  interpreters  were  provided 
with  maps,  sortie  plot  overlays,  and  standard  references  or  photo  keys 
for  each  performance  measure.  They  were  also  given  situation  sheets 
showing  the  number  of  photographs  included  in  the  test,  the  scale  of 
the  photos,  the  specific  request  for  information,  and  the  general 
battlefield  situation  at  the  time  the  photos  were  obtained.  The 
situation  sheets  for  the  three  performance  measures  are  given  in 
Appendix  B. 


EXPERIMENTAL  APPROACH 

This  experiment  was  designed  to  determine  whether,  for  a  given 
interpretation  requirement,  interpreter  performance  would  vary  as  a 
function  of  different  methods  of  stating  the  requirement.  For  each 
requirement,  a  different  performance  measure  was  used,  T-17  for  the 
completeness  requirement,  T-10  for  accuracy  and  completeness,  and  T-23 
for  accuracy. 

The  design  thus  required  tlu-ee  single-factor  experiments,  one 
experiment  for  each  requirement.  Each  of  the  tlu-ee  groups  was  assigned 
to  a  method  for  a  particular  test.  Each  group  was  also  administered  the 
tlu-ee  performance  measures  in  the  same  order  in  accordance  with  the 
assigned  experimental  method  (See  Appendix  C  for  Raw  Score  Distribution). 

Immediately  prior  to  data  collection,  the  situation  sheet  for  a 
given  measure  was  first  read  aloud  to  the  subjects.  After  answering 
any  questions  raised  by  the  subjects  concerning  their  assignments,  the 
military  situation  and  the  specific  method  under  which  the  subjects  in 
each  group  were  to  make  their  identifications  was  read.  The  subjects 


then  began  interpreting  the  imagery.  The  subjects  were  allowed  50, 
bO,  and  25  minutes  to  complete  measures  T-10,  T-17,  and  T-25,  respec- 

tlvely. 


VARIABLES 

Tlie  main  concern  of  this  experiment  was  to  determine  the  effect  of 
the  methods  of  stating  interpretation  requirements  on  interpreter 
performance.  The  effect  of  the  experimental  methods  employed  for  each 
requirement  was  determined  for  four  dependent  variables . 

1.  Number  of  targets  correctly  identified  (R) 

2.  Number  of  targets  incorrectly  identified  (w) 

5.  Accuracy  of  identification:  -t---- . , 

(R  +  W) 

Weighted  accuracy  score  (XR-YW)  where  X  equals  the  positive 
value  assigned  to  each  right  score,  and  Y  equals  the  negative  value 
assigned  to  each  wrong  score  for  each  requirement. 

The  weighted  scores  were  included  to  determine  the  effects  of 
tlie  tliree  methods  where  performance  was  actually  scored  in  terms  of  the 
values  given  either  explicitly  or  implicitly  in  the  instructions , 
depending  upon  the  method  used.  The  weighted  scores  were  computed  using 
the  values  assigned  to  tlie  payoff  method  (See  Table  1)  and  weighting 
the  raw  scores  for  all  methods  accordingly.  It  was  assumed  tliat  the 
instructions  were  fairly  comparable  across  all  methods  for  each  require¬ 
ment.  Although  the  values  used  were  explicit  for  the  payoff  method,  to 
a  degree  suiiicient  tor  this  exploratory  study,  they  were  implied  in 
the  instructions  for  the  other  methods. 

The  specific  weights  were  assigned  as  follows: 

Image  Interpretation  Requirement 

Weights  Accuracy  Completeness 

Rights  +1  +1 

Wrongs  -9  0 


RESULTS 

Table  2  shows  the  mean  mumber  of  correct  and  incorrect  identifica¬ 
tions  as  veil  as  the  mean  accuracy  obtained  under  the  completeness 
requirement.  As  is  readily  apparent  from  the  consistency  among  the 
means  of  the  different  variables,  none  of  the  dependent  variables 
yielded  significant  differences.  Tills  finding  is  consistent  with 
expectations  since  under  all  of  the  methods  the  interpreter  is  "encouraged" 
to  make  all  the  identifications  he  possibly  can  without  fear  of  impunity. 


Accuracy  !■ 
Completeness 

+1 

-1 
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Table  2 

MEAN  performance  indices  for  three  methods  of  stating 
THE  COMPLETENESS  REQUIREMENT  (T-17) 


Method 

Mean  No. 
Correct 

Mean  No. 
Incorrect 

Mean 

Accuracy 

Payoff 

39-8 

21.9 

65 

Certitude 

37-5 

24.9 

5 8 

Verbal 

40.3 

24.7 

62 

Table  3  shows  the  same  kind  of  data  obtained  under  the  accuracy 
and  completeness  requirement.  As  for  the  completeness  requirement,  the 
number  of  correct  and  incorrect  identif ications  did  not  yield  signifi¬ 
cant  differences.  However,  the  mean  accuracy  of  identification  did  vary 
significantly  among  the  three  methods. 

Table  3 

MEAN  PERFORMANCE  INDICES  FOR  THREE  METHODS  OF  STATING 
THE  ACCURACY  AND  COMPLETENESS  REQUIREMENT  (T-IO) 


Method 

Mean  No. 
Correct 

Mean  No. 
Incorrect 

Mean 

Accuracy* 

Payoff 

2.6 

9-5 

27 

Certitude 

2.9 

11.9 

51 

Verbal 

1.9 

11.3 

14 

*  Difference  in  mean  accuracy  significant  at  P  <.01 


Review  of  table  3  shows  that  the  mean  accuracy  for  the  certitude 
method  was  approximately  3*5  times  higher  tlian  that  of  the  verbal  method, 
and  a  little  less  than  twice  that  of  the  payoff  method.  The  mean  accura¬ 
cy  of  the  certitude  method  was  51jfc.  It  might  be  expected  that  for  this 
mean  accuracy,  a  smaller  mean  number  of  incorrect  identifications  would 
have  been  obtained.  However,  the  size  of  the  mean  number  of  incorrect 
identifications  obtained  for  the  certitude  method  is  due  primarily  to 
the  responses  of  two  interpreters  who  were  responsible  for  90  of  the 
119  incorrect  identifications  made.  Without  such  poor  perf ormance ,  the 
mean  number  of  incorrect  identifications  made  under  the  certitude  method 
would  undoubtedly  liave  been  significantly  smaller  and  also  significantly 
different  from  the  other  means.  Such  widely  deviant  behavior  was  not 
evident  for  the  other  two  methods  (See  Appendix  C).  However,  since  mean 
accuracy  was  based  on  the  mean  of  the  individual  accuracy  scores  and  not 
on  the  total  right  and  wrong  identifications  made  by  the  group,  the  score 


did  not  penalize  the  whole  group  for  the  adverse  performance  of  a  small 
number  of  individuals  within  the  group. 

The  low  mean  accuracy  of  the  verbal  method  may  be  attributed  to  the 
conflict  engendered  by  the  interpreter 's  attempting  to  perform  under  the 
conflicting  requirements  of  accuracy  and  completeness.  The  low  mpp n 
accuracy  of  the  payoff  method  is  a  little  puzzling.  It  is  possible  that 
the  values  of  the  awards  (+  l)  end  penalties  (-1)  which  were  used  were 
not  large  enough  to  induce  the  desired  behavior.  However,  there  are  no 
data  to  guide  the  experimenter  In  the  assignment  of  values  for  correct 
and  incorrect  performance  to  achieve  the  desired  end.  The  study  is 
therefore  Inconclusive  with  respect  to  the  ability  of  the  payoff  method 
to  yield  information  with  the  desired  accuracy  and  completeness  attri¬ 
butes.  This  can  only  be  done  by  means  of  systematic  research  to  deter¬ 
mine  which  values  function  appropriately  for  a  given  military  situation. 

The  analyses  of  variance  failed  to  produce  significant  differences 
in  interpreter  performance  under  the  accuracy  requirement.  However,  as 
shown  in  table  4,  there  does  appear  to  be  a  trend  in  the  means  of  the 
accuracy  with  which  the  targets  were  identified.  This  trend  favors  the 
payoff  method,  followed  in  turn  by  the  certitude  and  verbal  methods. 

It  is  possible  that  the  reason  the  mean  accuracy  of  the  certitude 
group  was  not  higher  was  the  considerable  frequency  with  which  inter¬ 
preters  tend  to  overestimate  their  confidence  in  the  identifications 
which  they  make. 


Table  4 


MEAN  PERFORMANCE  INDICES  FOR  THREE  METHODS  OF  STATING 
THE  ACCURACY  REQUIREMENT  (T-23) 


Method 

Mean  No. 
Correct 

Mean  No. 
Incorrect 

Mean 

Accuracy 

Payoff 

2.6 

1.3 

77 

Certitude 

3.0 

2.7 

63 

Verbal 

2.2 

3-2 

U6 

The  analysis  of  variance  for  the  weighted  accuracy  scores  for 
accuracy  and  completeness  requirement  and  for  the  accuracy  requirement 
failed  to  produce  any  significant  results.  Thus,  it  appeal’s  that  the 
different  methods  of  stating  the  requirements  had  little  effect  upon 
interpreter  performance  where  such  performance  was  evaluated  in  terms 
of  quantitative  values  given,  explicitly  or  implicitly,  in  the  instruc¬ 
tions.  The  mean  weighted  scores  (Table  5)  were  negative  indicating 
that  the  interpreters  did  not  conform  to  the  instructions  in  making 
their  identifications. 


MEAN  WEIGHTED  SCORES  FOR  THREE  METHODS  OF  STATING  THE 
ACCURACY  AND  COMPLETENESS  (T-IO)  AND 
ACCURACY  (T-23)  REQUIREMENTS 


Method 

INTERPRETATION  REQUIREMENTS 

A.ccuracy  and  Complete¬ 
ness  (R  =  +1,  W  =  -l) 

Accuracy 
(R  =  +1,  W  =  -9) 

Payoff 

-9.1 

-6.7 

Certitude 

-11.3 

-9-0 

Verbal 

-16.6 

-9-9 

' 

Only  two  analyses  of  variance  were  computed,  one  for  the  accuracy  and 
completeness  requirement,  and  one  for  the  accuracy  requirement.  The 
weighted  scores  for  the  completeness  requirement  were  the  same  as  the 
unweighted  scores  used  in  the  unweighted  analysis  (X  =  1,  Y  =  0); 
consequently,  no  additional  analysis  was  required.  Even  though  the 
differences  in  weighted  scores  were  not  significant  for  either  require¬ 
ment,  the  mean  weighted  scores  for  the  payoff  method  were  the  highest 
for  each  requirement  followed  in  order  by  those  for  the  certitude 
method  and  the  verbal  method.  All  analyses  of  variance  tables  are 
included  in  Appendix  E. 


DISCUSSION  AND  CONCLUSIONS 

The  main  purpose  of  the  experiment  was  to  determine  whether  inter¬ 
preter  perfonnance  was  affected  by  presenting  intelligence  requirements 
for  accuracy,  completeness,  and  accuracy  and  completeness  each  in  three 
different  ways.  Interpreter  performance  was  scored  in  two  ways.  In 
addition  to  the  three  indices  routinely  obtained- -number  of  correct 
identifications,  number  of  incorrect  identifications,  and  mean  accuracy 
of  identifications — a  fourth  index  was  used,  a  weighted  score  in  which 
the  weights  were  the  values  assigned  to  rights  and  wrongs  scores  under 
the  payoff  method  of  presenting  information  requirements. 

Nonsignificant  results  were  obtained  for  the  three  regular  indices 
of  interpreter  performance  when  the  interpreter  was  operating  under  two 
of  the  three  requirements,  accuracy  and  completeness.  For  the  weighted 
scores  also,  results  were  nonsignificant  for  accuracy  and  for  the 
accuracy  and  completeness  combined.  However,  the  significant  result 
for  mean  accuracy  of  identifications  under  the  combined  accuracy  and 
completeness  requirement  offers  some  hope  for  the  certitude  method  as 
possibly  useful  for  stating  intelligence  requirements,  particularly 
where  conflicting  requirements  for  conqpleteness  and  accuracy  are 
imposed  simultaneously.  As  noted  earlier,  the  additional  important 


requirements  for  timeliness  and  level  of  reporting  detail  were  not 
included  in  the  study.  Inclusion  of  these  requirenu  at.  in  future  research 
may  veil  influence  the  direction  of  tne  results  of  studies  such  as  this. 

Fast  research  lias  shown  that  the  interpreter  consistently  overesti¬ 
mates  his  confidence  in  his  identifications.  How eve",  with  additional 
training  estimating  his  confidence,  it  is  postulated  tluit  the  inter¬ 
preter's  expressed  Judgment  of  the  probability  of  his  identification 
being  correct  may  be  more  closely  related  with  the  accuracy  of  his 
identifications.  Should  this  be  the  case,  then  the  certitude  method 
should  certainly  be  one  of  the  promising  methods  to  be  considered  in 
futiu'c  research. 


The  payoff  method  also  shows  promise  as  a  method  of  stating  intel¬ 
ligence  requirements .  The  relative  poor  performance  obtained  for  the 
unweighted  scores  under  accuracy  and  completeness  requirement  mai  be  a 
function  of  the  values  employed .  Under  the  accuracy  requirement,  this 
method  yielded  the  highest  mean  accuracy  of  the  three  methods,  und  Just 
missed  being  significantly  different  from  the  other  methods.  Although 
the  means  were  not  significantly  uifferent  for  the  weighted  scores, 
there  was  a  trend  for  the  payoff  method  to  liave  the  highest  scores 
under  the  accuracy  and  completeness  and  accuracy  requirements .  Further 
research  using  different  values  may  provide  more  significant  findings. 
Further,  it  is  possible  that  a  combination  of  the  payoff  and  certitude 
methods  may  serve  ns  a  potential  optimal  method,  since  for  some  require¬ 
ments  the  payoff  method  -may  be  more  productive,  and  for  others  the 
certitude  method.  Future  research  will  explore  this  possibility. 

Generally  speaking,  the  verbal  description  of  the  requirements 
appeared  to  be  the  least  effective  of  the  tlu'ec  methods.  Such  perform¬ 
ance  for  the  completeness  requirement  and  for  the  accuracy  requirement 
could  be  cave  to  the  lack  of  a  standard  frame  of  reference  generally 
applicable  to  each  of  the  requirements,  and  also  to  different  connota¬ 
tions  engendered  among  different  interpreters  regarding  the  precise 
meaning  of  each  of  the  requirements,  rive  poor  performance  under  the 
accuracy  and  completeness  requirement  typifies  the  results  obtained 
when  interpreters  work  under  simultaneously'  imposed  conflicting  require¬ 
ments. 


Note  thnt  completeness  was  a  dependent  variable  tluit  required 
analysis  in  the  contextual  setting  of  this  study.  Since  the  complete¬ 
ness  score  normally  used  is  a  ratio  of  rights  over  rights  plus  wrongs 
plus  misidentif ications  ^ where  rights  plus  wrongs  plus  misidentifioations 
equal  h  or  the  total  number  of  targets  in  the  performance  measure  and 
is  a  constant  for  all  subjects  for  a  given  performance  measure ) ,  the 
number  of  rights  is  a  perfect  correlate  of  completeness.  Thus,  only 
number  of  rights  was  used  in  the  study. 


However,  in  some  information  requirements  an  item  of  concern  is  a 
detection  accuracy  score  which  is  a  function  of  the  complement  of  the 
number  of  omits.  A  ratio  representing  this  score  would  be  expressed 
us  a  ratio  of  rights  und  misidentifioations  over  K  where  K  equals  the 
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total  number  of  targets  in  the  scoring  key.  Although  an  analysis  of 
this  variable  was  not  conducted  in  this  exploratory  investigation,  it 
will  be  included  as  one  of  the  more  important  variables  in  a  more 
comprehensive  study. 

The  major  implication  of  the  experiment  is  that  different  methods 
of  stating  requirements  liave  an  effect  on  performance,  and  that  the 
certitude  method  (i.e.  requiring  the  interpreter  to  identify  only  those 
targets  for  which  his  level  of  confidence  is  5^  or  better)  appears  to 
be  one  promising  method  of  stating  requirements  when  conflicting 
requirements  of  completeness  and  accuracy  are  imposed  simultaneously.  t 
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A-l. 


INSTRUCTIONS  FOR  COMFLETENEOS  REQUIREMENT 


General  Military  Situation 

Higher  headquarters  is  planning  an  a] rborne  drop  of  a  battle 
group  in  an  area  adjacent  to  your  zone  of  reaponaihility.  Due  to  the 
nature  of  the  mission,  all  enemy  objects  munt  be  reported.  Before  the 
drop  zone  is  selected  all  possible  targets  should  be  located  and 
identified.  Your  mission  is  to  locate  and  identify  all  possible 
military  targets. 

Instructions  for  Method  A  (Payoff  Method) 

Your  performance  will  be  evaluated  as  follows:  You  will  receive 
one  point  for  each  correct  identification.  Errors  will  not  count 
against  you.  Your  total  score  therefore  depends  entirely  on  the  number 
of  correct  identifications  you  make. 

Instructions  for  Method  B  (Certitude  Method) 

Therefore,  identify  all  possible  military  targets  oven  though 
your  level  of  confidence  in  your  Identifications  may  be  near  zero . 

Instructions  for  Method  C  (Verbal  Method) 

Therefore,  be  as  complete  as  possible  in  your  identifications 
and  identify  all  possible  military  targets. 
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A-.'.  INUTRUCTrONU  FOR  ACCURACY  AND  COtlFLETENKSS  REQUIREMENT 


An  armored  attack  la  being  planned  for  the  area  of  your  .'.one  of 
responsibility ,  it  is  important  Uuit  all  areas  of  enemy  military  activ- 
tty  be  located. 

Your  mission  is  to  locate  and  identity  all  military  targets,  but 
at  the  same  time,  try  not  to  make  any  erroneous  identifications. 

Instructions  for  Method  A  (Payoff  Method) 

To  stress  the  importance  of  maintaining  a  balance  between  accuracy 
and  completeness  your  performance  will  be  evaluated  as  follows:  You 
will  get  one  point  for  each  correct  identification,  however,  you  will 
also  lose  a  point  for  each  error  you  make. 

Instructions  for  Method  1'  (Certitude  Method) 

Therefore,  only  Identity  those  objects  for  which  your  level  of 
confidence  in  your  identification  is  'xty>  or  better. 

Instructions  for  Method  C  (Verbal  Method) 

Therefore,  be  as  complete  as  possible  in  your  identifications,  but 
at  the  same  time  be  as  accurate  as  possible. 
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INSTRUCTIONS  FOR  ACCURACY  REQUIREMENT 


General  Military  Situation 

The  U.  S.  military  units  opposing  enemy  units  in  your  zone  of 
responsibility  are  under  strength  and  low  on  supplies,  particularly 
ammunition.  To  avoid  possible  waste  of  ammunition  and  other  supplies 
priority  must  be  given  to  those  objects  which  can  be  clearly  identified. 

Your  mission  is  to  locate  and  identify  only  those  objects  which 
you  are  certain  contain  military  targets. 

Method  A  (Payoff  Method) 

To  stress  the  importance  of  avoiding  errors  your  performance  will 
be  evaluated  as  follows:  You  will  get  one  point  for  each  correct 
identification,  but  you  will  lose  9  points  for  each  incorrect  identifi¬ 
cation. 

Method  B  (Certitude  Method) 

Therefore,  only  identify  those  objects  for  which  your  level  of 
confidence  in  your  identification  is  SX>jb  or  better. 

Method  C  (Verbal  Method) 

Therefore,  be  very  accurate,  as  accurate  as  you  can  possibly  be, 
in  making  your  identifications. 


B-l.  PERSONNEL  RESEARCH  BRANCH 

IMAGE  INTERPRETATION  PERFORMANCE  TEST 

PACKET  NR:  T-10 

CONTENTS:  Situation  Sheet,  answer  sheet,  photos  8  to  14  from  Mission 
R474B  16  August  1950,  map  sheet  6822  II  (INDONG,  scale 
1:50,000),  sortie  plot  overlay,  objects  list 


GENERAL  SITUATION: 

You  are  a  member  of  the  25th  Infantry  Division  PI  team  in  Korea  in 
August  1950.  The  NKPA  has  increased  pressure  both  north  and  east  of 
TAEGU.  The  front  now  extends  from  KWAN-DONG  to  HAKSONG-DONG  in  your 
division  sector,  (see  Map)  It  is  expected  that  the  enemy  soon  will 
attempt  major  attacks  in  this  area  in  order  to  secure  the  important 
transportation  center  of  TAEGU  southeast  of  your  present  positions. 


SPECIFIC  SITUATION: 

The  Division  G-2  Air  has  requested  that  your  PI  team  report  on  all 
enemy  activity  along  the  NAKTONG-GANG  (River).  Your  team  chief  has 
assigned  the  area  covered  by  photos  9  to  15  of  the  latest  photo  mission 
received.  He  has  determined  the  scale  to  be  approximately  1:7,000. 


REQUIREMENTS: 

Report  all  of  the  following  items: 

Vehicles 

Artillery 

Fortifications 

In  identifying  the  objects  on  your  answer  sheet  use  only  those  names 
appearing  on  the  attached  objects  list. 

You  have  50  minutes  to  complete  this  task. 


B-2  .  HUMAN  FACTORS  RESEARCH  BRANCH 

IMAGE  INTERPRETATION  PERFORMANCE  TEST 

PACKET  NR:  T-17 


CONTENTS:  Situation  Sheet,  Answer  Sheet,  Photos  1-7  from  Mission  1500 
F22,  and  Sortie  Plot 


GENERAL  SITUATION: 

The  date  is  29  October  i960.  You  are  an  image  interpreter  in  a  PI 
Company  attached  to  a  STRAC  unit  in  CONUS.  Hostile  forces  have  estab¬ 
lished  a  beachhead  on  the  eastern  coast  of  Florida  and  air  dropped 
special  forces  20  miles  inland.  The  air  dropped  special  forces  have 
disrupted  communications  and  captured  unknown  quantities  of  U.  S.  Army 
equipment . 


SPECIFIC  SITUATION: 

Hostile  forces  have  been  reported  in  the  area  covered  by  the 
photographs  in  your  packet. 


REQUIREMENTS: 

Report  all  military  objects  appearing  on  the  object  list  which 
you  can  locate  and  identify  on  the  aerial  photographs.  Use  only  those 
terms  appearing  on  your  object  list. 
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B-3.  U.  S.  ARM  PERSONNEL  RESEARCH  OFFICE 

WASHINGTON,  D.  C.  20315 

PERFORMANCE  MEASURE:  T-23  IMAGE  INTERPRETATION  TASK 


CONTENTS:  Situation  Sheet;  Immediate  Report  Form;  Photos  1037-1042 
from  Mission-  US  30-4488 


GENERAL  SITUATION: 

You  Eire  assigned  to  the  Immediate  Interpretation  Platoon  of  the 
Photo  Interpretation  Support  Unit  attached  to  the  US  First  Army.  The 
Allied  Forces  landed  on  the  Normandy  Peninsula  on  6  June  1944  and  for 
the  next  six  weeks  expanded  their  lodgement  area  in  preparation  for  a 
concentrated  attack.  The  breakout  was  successful  and  the  First  Army 
moved  beyond  St.  Lo.  Just  recently  the  Third  Army  attacked  South  and 
West,  securing  most  of  Brittany. 


SPECIFIC  SITUATION: 

Last  week  the  First  Army  easily  captured  Mortain  but  German  forces 
counter-attacked  successfully  two  days  ago.  The  situation  has  remained 
the  same  except  for  a  partially  successful  enemy  drive  in  their  northern 
sector  yesterday.  Your  unit  has  received  photographs  flown  three  hours 
ago  (scale  1:10,750)  and  artillery  in  the  area  of  enemy  penetrations 
has  been  given  top  priority.  You  have  been  assigned  part  of  the  photos 
received. 


REQUIREMENTS: 

Locate  and  identify: 


(Arty) 

Groups  of  3  or 
or  not 

(Tank) 

Concentrations 
on  the  road 

(Wheeled  V) 

Concentrations 
the  road 

(Wheeled  V- 

Concentrations 

road) 

road 

(Tank-road) 

Concentrations 
the  road 

more  FA  or  AA  positions  whether  occupied 
of  3  or  more  tanks  and/or  ;p  guns  not 
of  wheeled  vehicles  (3  or  more)  not  on 
of  wheeled  vehicles  (3  or  more)  on  the 
of  3  or  more  tanks  and/or  SP  guns  on 
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1.  A  concentration  of  tracked  or  wheeled  vehicles  is  defined  here 
as  3  or  more  occuring  within  on  area  of  about  250  by  250  meters  (approxi¬ 
mately  one  square  inch  on  your  photoglyphs ) . 

2.  A  concentration  of  tracked  or  wheeled  vehicles  on  the  road 
consist  of  a  group  of  3  or  more ,  each  being  within  one  inch  of  the 
preceding  vehicle. 

3.  Indicate  the  location  of  required  items  on  your  photograph 
by  placing  an  "X"  in  the  center  of  a  battery  or  in  the  center  of  a 
concentration  of  tracked  or  wheeled  vehicles  either  on  or  off  the  road. 

Do  not  report  individual  tracked  vehicles,  wheeled  vehicles,  or  guns. 
Report  only  concentrations  of  vehicles  and  batteries  and  indicate  the 
number  of  vehicles  and  guns  in  each. 

k.  Use  only  the  abbreviations  given  above  under  requirements  in 
identifying  the  objects  on  your  Spot  Report  Transmittal  Form. 

5-  You  will  have  25  minutes  to  complete  your  report. 


RAW  SCORE  DISTRIBUTIONS  OBTAINED  ON  T-10  UNDER 
THE  ACCURACY  AND  COMPLETENESS  REQUIREMENT 


Method  A 

■ 

1 

Method  B 

■ 

■ 

Method  C 

Man 

R 

w 

Acc 

Man 

R 

w 

Acc 

Man 

R 

w 

Acc 
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2 

0 

100 

11 

3 

0 

100 

21 

3 

7 

30 

2 

1 

8 

11 

12 

1 

0 

100 

22 

0 

8 

0 

5 

0 

10 

0 

13 

4 

0 

100 

23 

1 

7 

12 

4 

5 

4 

56 

14 

2 

1 

67 

24 

5 

18 

22 

5 

5 

8 

38 

15 

1 

48 

2 

25 

3 

10 

23 

6 

9 

23 

28 

16 

3 

9 

25 

26 

2 

11 

15 

7 

0 

16 

0 

17 

1 

10 

9 

27 

1 

15 

6 

8 

0 

4 

0 

18 

6 

3 

67 

28 

2 

7 

22 

9 

0 

14 

0 

19 

3 

6 

33 

29 

2 

24 

8 

10 

4 

6 

40 

20 

5 

42 

11 

30 

0 

11 

0 

£ 

26 

93 

273 

£ 

29 

119 

5l4 

£ 

19 

118 

138 

C-2.  RAW  SCORE  DISTRIBUTIONS  OBTAINED  ON  T-17  UNDER 
THE  COMPLETENESS  REQUIREMENT 


Method  A 

Method  B 

Method  C 

Man 

R 

w 

Acc 

Man 

R 

w 

Acc 

Man 

R 

w 

Acc 

1 

50 

12 

81 

11 

44 

33 

57 

21 

73 

34 

68 

2 

*+3 

17 

72 

12 

56 

15 

79 

22 

3^ 

12 

74 

3 

27 

33 

45 

13 

27 

24 

55 

23 

31 

10 

76 

4 

64 

15 

81 

14 

33 

22 

60 

24 

38 

49 

44 

5 

36 

41 

47 

15 

40 

28 

59 

25 

31 

22 

58 

6 

25 

6 

81 

16 

37 

22 

63 

26 

53 

19 

74 

7 

20 

24 

45 

17 

4 

28 

12 

27 

36 

30 

55 

8 

38 

10 

79 

18 

48 

21 

70 

28 

56 

29 

66 

9 

24 

19 

56 

19 

37 

15 

71 

29 

36 

13 

75 

10 

71 

42 

63 

20 

49 

41 

54 

30 

15 

29 

54 

£ 

219 

650 

£ 

375 

2*4-9 

~JjE~ 

£ 

*4-03 

247 

622 
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Method  A 


k'iV-  V»»kVJU  jJ 

Method  C 

Man 

Score 

Man 

Score 

11 

3 

21 

-4 

12 

1 

22 

-8 

13 

4 

23 

-6 

14 

1 

24 

-13 

15 

-47 

25 

-7 

1 6 

-6 

26 

-9 

17 

-9 

27 

-14 

18 

3 

28 

-5 

19 

-3 

29 

-22 

20 

-37 

30 

-11 

E  -90 

E  -99 

WEIGHTED  SCORE  DISTRIBUTIONS  OBTAINED  ON  T-23 
UNDER  THE  ACCURACY  REQUIREMENT 
(R  =  +1,  W  =  -9) 


E.  ANALYSIS  OF  VARIANCE  TABLES 


SUM  OF 

MEAN 

VARIABLE 

SOURCE 

SQUARES 

_  SQUARE 

F 

Completeness  Requirement  T-17 

Rights 

Between  Gps. 

44,60 

2 

22.30 

.086 

Within  Gps. 

6976.20 

27 

258.30 

Wrongs 

Between  Gps. 

56.27 

2 

28.14 

.228 

Within  Gps. 

3337.90 

27 

123.63 

Accuracy 

Between  Gps. 

263.47 

2 

131.74 

.509 

Within  Gps. 

6993.20 

27 

259.01 

Accuracy  and  Completenes 

s  Requirement 

T-10 

Rights 

Between  Gps. 

3.20 

2 

l.oO 

.829 

Within  Gps. 

52.00 

27 

1-93 

Wrongs 

Between  Gps. 

19-40 

2 

9-70 

1.820 

Within  Gps. 

143.80 

27 

5.33 

Accuracy 

Between  Gps. 

4864.07 

2 

2432.04 

2.49 

Within  Gps. 

26362.10 

27 

976.37 

Weighted 

Between  Gps. 

1610.60 

2 

805.30 

1.786 

Within  Gps. 

12171.40 

27 

450.30 

Accuracy  Requirement  T-23 

Rights 

Between  Gps. 

5.27 

2 

2.64 

•539 

Within  Gps. 

132.20 

27 

4.90 

Wrongs 

Between  Gps. 

43.40 

2 

21.70 

.164 

Within  Gps. 

3576.60 

27 

132.47 

Accuracy 

Between  Gps. 

7256.07 

2 

3628.04 

3-910 

Within  Gps. 

25072.10 

27 

928.60 

Weighted 

Between  Gps. 

5^.47 

2 

27.24 

.206 

Within  Gps, 

3573.00 

27 

132.33 

p(.Ol) 


* 
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